1. Field of the Invention
This invention relates to an image processing method and apparatus for transforming colors of an image, which is to be reproduced on, for example, a color print or a color display device. This invention also relates to a recording medium, on which a program for causing a computer to execute the image processing method has been recorded and from which the computer can read the program. This invention further relates to an image sensing system and an image reproducing system.
2. Description of the Related Art
With digital electronic still cameras (hereinbelow referred to as digital cameras), an image having been acquired with a photographing operation can be stored as a digital image signal on a recording medium, such as an internal memory or an IC card provided in the digital camera. The image having been acquired with the photographing operation can then be reproduced from the stored digital image signal as a hard copy, such as a print, or a soft copy on a display device. In cases where the image having been acquired with the digital camera is to be reproduced, it is desired that an image having image quality as good as the image quality of a photograph printed from negative film can be obtained.
Photographing operations with digital cameras are carried out under various photographing illuminants, such as tungsten light, fluorescent lamps, and outdoor daylight. Therefore, a digital camera, which has functions for automatically setting white balance in accordance with a photographing illuminant with respect to the image having been recorded with a photographing operation, has been proposed. Also, a technique, which utilizes characteristics such that a primary object pattern may be embedded at a center position in an image, and in which white balance is determined by dividing an image into subdivisions and assigning a large weight factor to a subdivision at the center position in the image, has been proposed (xe2x80x9cAutomatic White Balance Adjusting System Using Image Divisionxe2x80x9d by Kikuchi, et al., Technical Report of The Institute of Television Engineers of Japan, pp. 19-24, Jun. 26, 1990).
In cases where automatic correction is carried out by a digital camera in the manner described above, with respect to an image having been acquired under, for example, tungsten light, an adjustment is made such that the balance of RGB tristimulus values with respect to the tungsten light may become identical. The human visual system has functions referred to as chromatic adaptation such that, in both cases where white paper is seen under daylight and when the same white paper is seen under a fluorescent lamp having a spectral energy distribution different from the spectral energy distribution of the daylight, the color of the white paper can basically be recognized as white. However, in cases where the white paper is seen under tungsten light, the color of the white paper is often perceived as white with a yellow tint. Such phenomenon occurs since chromatic adaptation is not carried out completely. The phenomenon is referred to as incomplete chromatic adaptation. The incomplete chromatic adaptation is apt to occur when the tint of the viewing illuminant is strong or when the luminance of the viewing illuminant is low. Therefore, it is desired that colors of an image can be transformed by considering the chromatic adaptation and the incomplete chromatic adaptation of the human visual system, such that the reproduced image may have the same color appearances as the color appearances of the image, which was recorded with a photographing operation.
Therefore, an image processing method has been proposed wherein, after information representing photographing conditions with a digital camera, such as a photographing illuminant and brightness under which a photographing operation was carried out, has been recorded on a recording medium together with an image signal, the information representing the photographing conditions is fetched from the recording medium together with the image signal, image processing appropriate for the photographing conditions is carried out on the image signal, and an image true to the original scene is reproduced as a hard copy or a soft copy from the thus obtained processed image signal. Such an image processing method is proposed in, for example, Japanese Patent Application No. 9(1997)-207883.
However, the image processing method proposed in Japanese Patent Application No. 9(1997)-207883 is based upon the assumption that a digital camera is provided with the functions for recording the information representing the photographing conditions on the recording medium. Therefore, the proposed image processing method is not efficient for ordinary digital cameras.
With the aforesaid technique for automatically setting the white balance, it is necessary for a camera to be provided with means for automatically setting the white balance, and therefore the cost of the camera cannot be kept low. Ordinarily, cheap digital cameras are not provided with such means. With digital cameras which are not provided with the functions for automatically setting the white balance, in order for a reproduced image having good image quality to be obtained, it is necessary that an image is temporarily reproduced and displayed on a display device, and that an operator carries out an operation for adjusting the color balance and the white balance by seeing the displayed image. However, a great deal of skill is required to carry out such adjustment, and ordinary users cannot easily obtain satisfactory reproduced images by adjusting the white balance, and the like.
The primary object of the present invention is to provide an image processing method, wherein an image is capable of being transformed with a simple constitution and an easy operation such that a reproduced image giving an impression of the same appearances as the appearances of an original scene, which has been photographed, may be obtained.
Another object of the present invention is to provide an apparatus for carrying out the image processing method.
A further object of the present invention is to provide a recording medium, on which a program for causing a computer to execute the image processing method has been recorded and from which the computer can read the program.
A still further object of the present invention is to provide an image sensing system.
Another object of the present invention is to provide an image reproducing system.
The present invention provides a first image processing method, comprising the steps of:
i) calculating physiological primary colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, and
ii) carrying out image processing on the image signal and in accordance with the physiological primary colors having been calculated, a processed image signal being thereby obtained.
The term xe2x80x9cacquiring with an image sensing operationxe2x80x9d as used herein means that an object is photographed with a digital camera and an image signal representing the object image is thereby obtained, or that an object image, which has been recorded on negative film, reversal film, or the like, with a photographing operation, is read out from the film by using an image read-out apparatus, such as a scanner, and an image signal representing the object image is thereby obtained.
The present invention also provides a second image processing method, comprising the steps of:
i) transforming R, G, and B colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, into chromaticity values,
ii) calculating chromaticity mean values, which are mean values of the chromaticity values having been obtained with respect to the pixels in the image,
iii) transforming the chromaticity mean values into tristimulus values in accordance with physiological primary colors, and
iv) regarding the tristimulus values in accordance with physiological primary colors as the white of a photographing illuminant, under which the image signal was acquired, and transforming the image signal such that the tristimulus values may coincide with chromaticity values of a white point of a reference photographing illuminant, a processed image signal being thereby obtained.
The term xe2x80x9cphysiological primary colorsxe2x80x9d as used herein means the primary colors, which a person perceives with the visual cells of the eyes when he sees an object. It is considered that the phenomenon of white paper looking in white under various different illuminants is achieved by adjusting red, green, and blue response quantities (i.e., the tristimulus values), which are represented in accordance with physiological primary colors, under each of the various different illuminants.
The term xe2x80x9creference photographing illuminantxe2x80x9d as used herein means the photographing illuminant, which is different from the photographing illuminant used when an image signal is acquired, and which is to be used for reference when an image signal is to be acquired. Specifically, as for ordinary cameras, D50xcx9cD60 daylight is employed as the reference photographing illuminant.
The second image processing method in accordance with the present invention should preferably be modified such that, as the differences between the tristimulus values in accordance with physiological primary colors and the chromaticity values of the white point of the reference photographing illuminant become large, the degree of the coincidence may be kept to be low.
The present invention further provides a third image processing method, comprising the steps of:
i) transforming R, G, and B colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, into chromaticity values,
ii) calculating chromaticity mean values, which are mean values of the chromaticity values having been obtained with respect to the pixels in the image,
iii) transforming the chromaticity mean values into RGB tristimulus values, and
iv) regarding the RGB tristimulus values as the white of a photographing illuminant, under which the image signal was acquired, and transforming the image signal such that the RGB tristimulus values may coincide with chromaticity values of a white point of a reference photographing illuminant, a processed image signal being thereby obtained.
The third image processing method in accordance with the present invention should preferably be modified such that, as the differences between the RGB tristimulus values and the chromaticity values of the white point of the reference photographing illuminant become large, the degree of the coincidence may be kept to be low.
In the second and third image processing methods in accordance with the present invention, the chromaticity mean values should preferably be calculated as weighted mean values in accordance with a reciprocal of a distance of the chromaticity values from a CIE daylight locus or a black body locus.
The term xe2x80x9cCIE daylight locusxe2x80x9d as used herein means the locus representing the daylight color on the CIE chromaticity diagram. The term xe2x80x9cblack body locusxe2x80x9d as used herein means is the locus representing the relationship between the absolute temperature of a black body and the chromaticity at the absolute temperature. The CIE daylight locus and the black body locus are the ones utilized ordinarily.
The term xe2x80x9cweighting in accordance with a reciprocal of a distancexe2x80x9d as used herein means that, as the distance between the chromaticity values of a pixel and the daylight locus or the black body locus becomes large, the degree of weighting is set to be low.
Also, in the second and third image processing methods in accordance with the present invention, the white point of the reference photographing illuminant should preferably be set in accordance with the white point of the photographing illuminant such that the white point of the reference photographing illuminant may have values falling within a predetermined range.
The term xe2x80x9csetting such that a white point may have values falling within a predetermined rangexe2x80x9d as used herein means that, instead of being set as only one point, the white point of the reference photographing illuminant is set so as to have values falling within a certain range.
In the first, second, and third image processing methods in accordance with the present invention, the processed image signal should preferably be transformed even further in accordance with reproduction target values of a reproducing apparatus, which reproduces a visible image from the processed image signal.
The present invention still further provides a first image processing apparatus, comprising:
i) a physiological primary color calculating means for calculating physiological primary colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, and
ii) an image processing means for carrying out image processing on the image signal and in accordance with the physiological primary colors having been calculated, a processed image signal being thereby obtained.
The present invention also provides a second image processing apparatus, comprising:
i) a chromaticity value transformation means for transforming R, G, and B colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, into chromaticity values,
ii) a chromaticity mean value calculating means for calculating chromaticity mean values, which are mean values of the chromaticity values having been obtained with respect to the pixels in the image,
iii) a tristimulus value transformation means for transforming the chromaticity mean values into tristimulus values in accordance with physiological primary colors, and
iv) a chromatic adaptation transformation means for regarding the tristimulus values in accordance with physiological primary colors as the white of a photographing illuminant, under which the image signal was acquired, and transforming the image signal such that the tristimulus values may coincide with chromaticity values of a white point of a reference photographing illuminant, a processed image signal being thereby obtained.
In the second image processing apparatus in accordance with the present invention, the chromatic adaptation transformation means should preferably operate such that, as the differences between the tristimulus values in accordance with physiological primary colors and the chromaticity values of the white point of the reference photographing illuminant become large, the degree of the coincidence may be kept to be low.
The present invention further provides a third image processing apparatus, comprising:
i) a chromaticity value transformation means for transforming R, G, and B colors of each of pixels in an image, which is represented by an image signal having been acquired with an image sensing operation, into chromaticity values,
ii) a chromaticity mean value calculating means for calculating chromaticity mean values, which are mean values of the chromaticity values having been obtained with respect to the pixels in the image,
iii) a tristimulus value transformation means for transforming the chromaticity mean values into RGB tristimulus values, and
iv) a chromatic adaptation transformation means for regarding the RGB tristimulus values as the white of a photographing illuminant, under which the image signal was acquired, and transforming the image signal such that the RGB tristimulus values may coincide with chromaticity values of a white point of a reference photographing illuminant, a processed image signal being thereby obtained.
In the third image processing apparatus in accordance with the present invention, the chromatic adaptation transformation means should preferably operate such that, as the differences between the RGB tristimulus values and the chromaticity values of the white point of the reference photographing illuminant become large, the degree of the coincidence may be kept to be low.
In the second and third image processing apparatuses in accordance with the present invention, the chromaticity mean value calculating means should preferably calculate the chromaticity mean values as weighted mean values in accordance with a reciprocal of a distance of the chromaticity values from a CIE daylight locus or a black body locus.
Also, the second and third image processing apparatuses in accordance with the present invention should preferably further comprise means for setting the white point of the reference photographing illuminant in accordance with the white point of the photographing illuminant such that the white point of the reference photographing illuminant may have values falling within a predetermined range.
The first, second, and third image processing apparatuses in accordance with the present invention should preferably further comprise an output transformation means for transforming the processed image signal in accordance with reproduction target values of a reproducing apparatus, which reproduces a visible image from the processed image signal.
Each of the first, second, and third image processing methods in accordance with the present invention may be furnished as a program for causing a computer to execute the image processing method and having been recorded on a recording medium, from which the computer can read the program.
The present invention still further provides an image sensing system, comprising:
i) an image sensing means for detecting an object image with an image sensing operation and thereby obtaining an image signal, which represents the object image,
ii) an image processing apparatus in accordance with the present invention, which carries out image processing on the image signal and thereby obtains a processed image signal, and
iii) a recording means for recording the processed image signal, which has been obtained from the image processing carried out by the image processing apparatus, on a recording medium.
The present invention also provides an image reproducing system for reproducing a visible image from an image signal, comprising:
i) an input means for receiving the image signal,
ii) an image processing apparatus in accordance with the present invention, which carries out image processing on the image signal having been received from the input means and thereby obtains a processed image signal, and
iii) a reproducing means for reproducing the visible image from the processed image signal, which has been obtained from the image processing carried out by the image processing apparatus.
With the first image processing method and apparatus in accordance with the present invention, the image processing is carried out on the image signal and in accordance with the physiological primary colors of each of the pixels in an image, which is represented by the image signal. Therefore, the image signal is processed in accordance with the colors, which are recognized by a person when he sees the object. Accordingly, the image processing can be carried out on the image signal, such that an image true to an impression, which a person received when he saw the object image, may be reproduced. From the processed image signal having been obtained from the image processing, an image true to the impression, which a person received when he saw the object image, can be reproduced.
With the second image processing method and apparatus in accordance with the present invention, the R, G, and B colors of each of the pixels in an image are transformed into the chromaticity values, and the chromaticity mean values, which are the mean values of the chromaticity values having been obtained with respect to the pixels in the image, are calculated. The chromaticity mean values are then transformed into the tristimulus values in accordance with physiological primary colors. In the second image processing method and apparatus in accordance with the present invention, the tristimulus values in accordance with physiological primary colors, which tristimulus values have been calculated from the image signal, are regarded as the white which a person perceives when he sees the object at the site, i.e., the white of the photographing illuminant, under which the image signal was acquired. The image signal is transformed such that the tristimulus values in accordance with physiological primary colors may coincide with the chromaticity values of the white point of the reference photographing illuminant. The appearances of an image vary for different photographing illuminants, under which the image signal is acquired. For example, in cases where an image is photographed in a cloudy day or in the shade, the white in the image looks in white with a blue tint as a whole. However, by virtue of the chromatic adaptation, a person recognizes the white with a blue tint as the white. Therefore, with the second image processing method and apparatus in accordance with the present invention, wherein the image signal is transformed such that the tristimulus values in accordance with physiological primary colors may coincide with the chromaticity values of the white point of the reference photographing illuminant, the white represented by the image signal is transformed into the same white as the white, which is represented by an image signal acquired under the reference photographing illuminant. As a result, an image true to the impression of the image appearances at the time of the acquisition of the image signal can be reproduced.
The second image processing method and apparatus in accordance with the present invention may be modified such that, as the differences between the tristimulus values in accordance with physiological primary colors and the chromaticity values of the white point of the reference photographing illuminant become large (as in cases where white paper is seen under tungsten light), the degree of the coincidence between the tristimulus values and the chromaticity values of the white point of the reference photographing illuminant may be kept to be low. In such cases, the image signal can be transformed such that the feeling given under the photographing illuminant at the time of the photographing operation may remain. In this manner, an image true to the appearances of the original scene can be reproduced.
With the third image processing method and apparatus in accordance with the present invention, the R, G, and B colors of each of the pixels in an image are transformed into the chromaticity values, and the chromaticity mean values, which are the mean values of the chromaticity values having been obtained with respect to the pixels in the image, are calculated. The chromaticity mean values are then transformed into the RGB tristimulus values. Also, as in the second image processing method and apparatus in accordance with the present invention, the RGB tristimulus values are regarded as the white of the photographing illuminant, under which the image signal was acquired, and the image signal is transformed such that the RGB tristimulus values may coincide with the chromaticity values of the white point of the reference photographing illuminant. Therefore, the white represented by the image signal is transformed into the same white as the white, which is represented by an image signal acquired under the reference photographing illuminant. As a result, an image true to the impression of the image appearances at the time of the acquisition of the image signal can be reproduced. Also, in the aforesaid second image processing method and apparatus in accordance with the present invention, wherein the chromatic adaptation transformation is carried out in accordance with the tristimulus values in accordance with physiological primary colors, it is necessary for the image signal to be transformed into the physiological primary colors at the time of the chromatic adaptation transformation. However, with the third image processing method and apparatus in accordance with the present invention, wherein the chromatic adaptation transformation is carried out in accordance with the RGB tristimulus values, it is not necessary for the image signal to be transformed into the physiological primary colors at the time of the chromatic adaptation transformation. Accordingly, with the third image processing method and apparatus in accordance with the present invention, the time required to carry out the operations can be kept short, and the constitution of the image processing apparatus can be kept simple.
Further, the third image processing method and apparatus in accordance with the present invention may be modified such that, as the differences between the RGB tristimulus values and the chromaticity values of the white point of the reference photographing illuminant become large (as in cases where white paper is seen under tungsten light), the degree of the coincidence between the RGB tristimulus values and the chromaticity values of the white point of the reference photographing illuminant may be kept to be low. In such cases, the image signal can be transformed such that the feeling given under the photographing illuminant at the time of the photographing operation may remain. In this manner, an image true to the appearances of the original scene can be reproduced.
On the chromaticity diagram, chromaticity values, which are located at a position remote from the CIE daylight locus or the black body locus, may be regarded as not being the color of the illuminant and as being an object color, such as the color of red clothes or the blue sky. Therefore, in cases where the chromaticity mean values are calculated as the weighted mean values in accordance with the reciprocal of the distance of the chromaticity values from the CIE daylight locus or the black body locus, the tristimulus values in accordance with physiological primary colors or the RGB tristimulus values can be obtained such that they may more accurately represent the white of the photographing illuminant. Accordingly, in cases where the tristimulus values are calculated from the thus obtained chromaticity mean values, more accurate image processing can be carried out.
In cases where the white point of the reference photographing illuminant is set in accordance with the white point of the photographing illuminant such that the white point of the reference photographing illuminant may have values falling within a predetermined range, processing can be carried out such that the color of the photographing illuminant, under which the image signal was acquired, may be reflected. Also, since the white point of the reference photographing illuminant has values falling within the predetermined range, an appropriate white point falling within the predetermined range and the tristimulus values in accordance with physiological primary colors or the RGB tristimulus values can be made to coincide with each other. In this manner, the advantages over the cases where the tristimulus values are made to coincide with only one white point can be obtained in that excessive chromatic adaptation transformation can be prevented from being carried out. As a result, a processed image signal, from which an image having a more natural feeling can be reproduced, can be obtained.
Further, the processed image signal has been corrected such that the white point represented by the processed image signal may coincide with the white point obtained under the reference photographing illuminant. Therefore, in cases where the processed image signal is transformed even further in accordance with the reproduction target values of the reproducing apparatus, the processed image signal can be transformed such that appropriate reproduction target values may be achieved. Accordingly, a reproduced image having good image quality can be obtained from the processed image signal having thus been transformed.
With the image sensing system in accordance with the present invention, which is provided with the image processing apparatus in accordance with the present invention, the image signal having been acquired with the image sensing operation, is recorded on the recording medium as the processed image signal, from which the impression of the appearances at the time of the image sensing operation can be reproduced. Therefore, an image signal, from which the image true to the impression of the appearances at the time of the image sensing operation can be reproduced, can be obtained by merely carrying out the image sensing operation on the object.
With the image reproducing system in accordance with the present invention, which is provided with the image processing apparatus in accordance with the present invention, the inputted image signal is subjected to the image processing such that the impression of the appearances at the time of the photographing operation can be reproduced. The thus obtained processed image signal is then used to reproduce a visible image. Therefore, an image true to the impression of the appearances at the time of the photographing operation can be reproduced by merely inputting the image signal.